The Effect of Facial Emotion and Action Depiction on Situated Language Processing

Two visual world eye-tracking studies investigated the effect of emotions and actions on sentence processing. Positively emotionally valenced German non-canonical object-verb-subject (OVS) sentences were paired with a scene depicting three characters (agent-patient-distractor) as either performing the action described by the sentence, or not performing any actions. These scene-sentence pairs were preceded by a positive prime in the form of a happy looking smiley (vs. no smiley) in experiment 1 and in the form of a natural positive facial expression (vs. a negative facial expression) in experiment 2. Previous research has demonstrated the effect of action depiction on sentence processing of German OVS sentences (Knoeferle, Crocker, Scheepers, & Pickering, 2005). Moreover, emotional priming facilitates sentence processing for older and younger adults (Carminati & Knoeferle, 2013). However, up to date there is no evidence as to whether schematic faces such as smileys are as effective as natural faces in facilitating sentence processing. These insights lead to the hypotheses that participants would not only profit from depicted events, but that processing of OVS sentences might also be positively affected by emotional cues. Plus, we assessed the degree of naturalness the emotional face needs to possess to affect sentence processing. Results replicate the predicted effect of action depiction (vs. no action depiction). The expected facilitatory effect of emotional prime is trending in both experiments. However, the effect is more pronounced in the natural face version (exp. 2) than in the smiley version (exp. 1).